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ABSTRACT 



This paper describes a new, but tried and tested, paradigm 
in teacher evaluation: the Three Ability Framework (3AF) . Previous research 
has identified three fundamental abilities that faculty implicitly expect to 
develop in their courses: technical skills, professional competence, and 
professional attitudes. The 3AF monitors the alignment of the student's and 
teacher 1 s expectations of these developments . The process respects the 
professional freedom of lecturers to be responsible for how they believe 
their subjects should be taught. Because the process is so efficient, it can 
be used two or three times during courses to monitor the quality of teaching. 
This evaluation method also links student assessment to the professional 
awareness of the teacher, resulting in a single number that is used as an 
administrative decision point. The 3AF is applied in a five-step process that 
begins with training provided the lecturer by the institution. Alignment 
between student ratings and the lecturer's own rating is calculated. It has 
been found in previous research that the alignment of percentage ratings 
correlates with academic attainment as measured by examination and coursework 
and that the alignment of proportions of the three abilities correlates with 
enjoyment of the course. (Contains 14 references.) (SLD) 
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SCOPE OF INTEREST NOTICE 



Overview 

1 .Traditional student 
assessment of teaching 
2.The3AF alternative 
IFive step application of 
the method 
4.Added advantages 



Traditional student 
assessment of teaching 
uses anonymous Likert 
opinionnaires given at 
end of course 



Common uses and 
associated problems 

- To give feedback on 
teaching (Likert average 

has little meaning for 
teaching) 

- For promotion and 
tenure decisions 
(anonymous forms have 
little legal credibility) 
- To improve course (it’s 
a post mortem 
assessment) 

- To assess customer 
satisfaction (inviting 

anonymous complaints 
undermines student/ 
teacher trust) 
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Traditional student evaluation of teaching is a servant that must serve four 
masters - the students, the administration, the teacher and staff development. 
Because these masters all want different things the servant seems doomed to 
failure. This is the case with the ubiquitous ‘opinionnaire’. Faculties have 
tried different questions, designs and processes but still the servant cannot 
do the job. This paper introduces a new, tried and tested paradigm in teacher 
evaluation - the Three Ability Framework (the 3AF). Previous research has 
identified three fundamental abilities that faculty implicitly expect to develop 
on their courses, to differing extents which depend on the subject and the 
level of the course. The 3 AF monitors the alignment of student’s and teacher’s 
expectations of these developments. The process is faster, more efficient 
and cost-effective than the traditional opinionnaire. It respects the professional 
freedom of lecturers to be responsible for how they believe their subjects 
should be taught and because the process is so efficient it can be used two or 
three times during courses to monitor the quality of teaching. The bonus to 
staff development is that this evaluation method also links student assessment 
with the professional awareness of the teacher. The method results in one 
single number that is used as an administrative decision point. 



A common form of student evaluation of faculty teaching has for some years been the 
Likert scale questionnaire (Seldin, 1984). This questionnaire is usually anonymous and 
contains questions that are meant to serve four purposes, those of the students, those of the 
teacher, those of staff development and those of the administration. Unfortunately, the process 
seems to be failing in all four respects Greenwald, 1997; Greenwald & Gillmore, 1997; 
Howard & Maxwell, 1982; Marsh & Dunkin, 1997; Marsh & Roche, 1997, 1998). The 
students expect that their feedback will improve their courses. However, the forms are 
completed at the end of their course so that any changes will not be to their course but for 
other students who follow on afterwards and for whom the changes many not be suitable. 

The teacher expects some feedback to help improve his or her course (Moses, 1996). The 
averages of Likert responses are statistically questionable and provide much less useful 
feedback than the three simple open-ended questions “What did you like?”, “What did you 
dislike?” “What can be improved and how?”. Staff development is also interested in 
improving the lecturer’s teaching but often finds that the forms devised for this purpose are 
purloined by administration for their purposes - (i) to rate the lecturer for purposes of tenure 
and promotion and (ii) to defuse potential discontent by giving students a voice (Askew, 
Brown, Rhodes, Wiliam & Johnson, 1997). This is also a failure. Universities are being 
advised that anonymous feedback has little courtroom credibility should faculty wish to 
contest administrative decisions based on them. Quality teaching and learning in higher 
education is a relationship between the student and the lecturer. At the moment the lecturer 
has little say over what students may enter into this relationship with him/her yet, through 
these anonymous feedback forms the lecturer must take the consequences if the relationship 
fails - perhaps through the one-sided non-contribution of some students. The overall effect 
on lecturer student relations of this adversarial process is professionally numbing (Arreola, 

1983; Cashin, 1983; Cherry, Grant, & Kalinos, 1988). Staff dumb down their teaching to 
boost students’ self-esteem, change examination essays to student popular multiple guess 
questions and lighten the workload to the lowest common denominator and so as to heighten 
their popularity ratings for the feedback. 

The 3 Af is a method of assessing the quality of teaching that avoids these problemsy. It was 
developed in an Australian University as part of a the process of evaluating Faculty 
Assessment-Processes. In-depth interviews with staff and students identified many- — 
assessment-based problems that decreased the quality of teaching and learning: such as the 
litigationary student, the student who wants to be told exactly what to do, the staff who 
assess attendance so students will turn up to lectures, students who so devalue the course 



that they cheat, etc. The 3Af arose from the need to solve these problems. It is a simple 
I process of ensuring that staff and students are working to the same expectations as described 
j by three basic abilities. It has been tested in clinical teaching trials and by computer simulation 
| (Bastick, 1995a). It has been successfully used in three universities and can be beneficially 
replicated in part or whole. 



The 3AF alternative 

- developed to avoid 
assessment problems 

- tested in controlled 
experiments (n=56) 

- successfully trialled 
in three universities ; 

- separates criteria of, 

effectiveness from j 
indicators of success 



Criteria of effective 

teaching: 

(i) maximize academic 
attainment 
(ii)maximize course 
enjoyment 



The three abilities are: 

Technical Skills. This is the traditional speed and accuracy of reproducing facts and processes 
and is assessed by timed accuracy of reproduction. 

Professional competence. This is the ability to use the skills in a novel situation or extend 
these skills in a novel way. The assessment is by justification of the appropriateness of what 
is done. 

Professional attitudes. These are the values that are appropriate to the subject. They are 
assessed by demonstration in practical situations. 

Previous research has indicated that faculty on professional courses expect students to move 
their emphasis from level I to level II to level III in different degrees that depend on the 
subject content and the maturity of the students (Bastick, 1995b). For example, lecturers 
would typically expect students in teacher training to be learning technical skills during the 
first year of their course, appropriately applying those skills to unique classroom situations 
in their second year and developing appropriate professional values on the final year of 
their course. Assessment problems arise because students do not have the same expectations 
as the staff - they may still be expecting to be shown what to do in the final year when the 
lecturer is trying to promote professional values. 



Indicator of effective 

teaching: 

Alignment of student 
and faculty 
expectations of the 
three abilities 
Construct validity 
p<0.01 for (i) and (ii) 



The 3AF is applied in a five-step process that starts with institutional staff development. 
Step 1 : In practice the institution trains the lecturer in how to teach and assess the three 
abilities and the lecturer has the responsibility of justifying to his or her peers the percentages 
of these three abilities that he/she considers most appropriate to the design and delivery of 
the course. 

Step 2: When it comes to teaching, the lecturer explains these three abilities to the students 
at the beginning of the course and explains how they will be taught and how the course 
assignments have been designed to offer opportunities for them to be assessed. Step 3: The 
teaching can be monitored on the course and assessed at the end of the course using the 
same form. This form asks students to rate each of the three abilities twice (i) how it is 
currently on the course and (ii) how it should be. The lecturer makes the same rating. Step 
4 is the processing of these forms: The change expected by the students is calculated by 
their rating of ‘how it should be’ minus their rating of ‘how it is\ The lecturer’s expectation 
of change is calculated in the same way. Step 5 is the calculation of alignment: The alignment 
of expectations is the difference between the lecturer’s expectations of change and the 
student’s expectation of change. Zero is the perfect score. This offers two alignment scores 
(i) an alignment of the percentages given (scope) and (ii) and alignment of the proportions 
of the three abilities. It has been found that the alignment of percentage ratings correlates 
p<0.01 with academic attainment measured by examination and coursework and that the 
alignment of the proportions of the three abilities correlates p<0.01 with enjoyment of the 
course. What makes this result of considerable practical application is that the success 
criteria of attainment and enjoyment are terminal but the success indicators of alignment 
can be used during the course. 



The feedback process takes as little a 10 minutes and so it can be used two or three times 
during a course to actually monitor the teaching on the course. It can also be used by the 
administration just before the end of the course to derive a single decision point number for 
the assessment of teaching. 



Added advantages 




Added advantages are that alignment can be calculated for any individual, or for minority 
students, or to compare if the teaching is more effective for one group than for another e.g. 
males v females, experienced v non-experienced students. 

d 
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